{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "543ff9ff",
   "metadata": {},
   "source": [
    "# 热点话题发现\n",
    "```\n",
    "聚类, 分类\n",
    "分类: 根据文本的特征或属性，划分到已有的类别中。这些类别是已知的，通过对已知分类的数据进行训练和学习，找到这些不同类的特征，再对未分类的数据进行分类。\n",
    "聚类: 不知道数据会分为几类，通过聚类分析(系统聚类法、有序样品聚类法、动态聚类法、模糊聚类法、图论聚类法、聚类预报法等)将数据或者说用户聚合成几个群体。\n",
    "\n",
    "k-means 聚类算法\n",
    "1.随机生成K个聚类中心(随机会导致复杂度较高, K不好调节)\n",
    "2.计算每个样本与每一个聚类中心的距离(余弦相似度),离哪个聚类中心近,就划分到哪个聚类中心所属的集合中\n",
    "3.重新计算每个集合的聚类中心\n",
    "4.重复2,3步直到收敛(聚类中心偏移很小,或者计算聚类中心次数超过阈值)\n",
    "5.返回所有的聚类标签(不知道有哪些标签,速度较慢)\n",
    "\n",
    "k-means++ 优化算法\n",
    "初始聚类中心之间的相互距离要尽可能的远(解决1对应的问题)\n",
    "single-pass 算法, 相近的样本(距离小于d)合并为一个聚簇 (解决5对应的问题)\n",
    "\n",
    "评价指标:\n",
    "高的类内(intra-cluster)相似度\n",
    "低的类间(inter-cluster)相似度\n",
    "-- 高内聚低耦合\n",
    "\n",
    "\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0fe481a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "array([[1, 0, 2],\n       [0, 0, 3],\n       [4, 5, 6]])"
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np \n",
    "import scipy\n",
    "from scipy.sparse.csgraph import connected_components\n",
    "\n",
    "\"\"\" \n",
    "稀疏矩阵\n",
    "\"\"\"\n",
    "#0表示默认起始点，0之后有几个数字就表示有几行\n",
    "indptr = np.array([0, 2, 3, 6]) # 行便宜\n",
    "indices = np.array([0, 2, 2, 0, 1, 2]) # 列位置\n",
    "data = np.array([1, 2, 3, 4, 5, 6]) # value\n",
    "\n",
    "scipy.sparse.csr_matrix((data, indices, indptr), shape=(3, 3)).toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d071503",
   "metadata": {},
   "source": [
    "## jieba 分词\n",
    "NLP（自然语言）领域现在可谓是群雄纷争，各种开源组件层出不穷，\n",
    "其中一支不可忽视的力量便是jieba分词，号称要做最好的 Python 中文分词组件\n",
    "\n",
    "+ **精确模式**：把文本精确的切分开，不存在冗余单词。\n",
    "+ **全模式**：把文本中所有可能的词语都扫描出来，有冗余。\n",
    "+ **搜索引擎模式**：在精确模式基础上，对长词再次切分。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a70575e",
   "metadata": {},
   "source": [
    "```bash\n",
    "pip install jieba\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1ea1ffd6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "全模式: #/ 高校/ 高校学生/ 学生/ 喊/ 楼/ EDG/ #/ /  / / 给/ 不/ 搞/ 电/ 竞/ 的/ 睡觉/ 人/ 一条/ 活路/ 行/ 吗/ ,/ EDG/ 牛/ 逼\n",
      "精确模式: #/ 高校学生/ 喊楼/ EDG/ #/  / 给/ 不/ 搞/ 电/ 竞/ 的/ 睡觉/ 人/ 一条/ 活路/ 行/ 吗/ ,/ EDG/ 牛/ 逼\n",
      "默认模式: #/ 高校学生/ 喊楼/ EDG/ #/  / 给/ 不/ 搞/ 电/ 竞/ 的/ 睡觉/ 人/ 一条/ 活路/ 行/ 吗/ ,/ EDG/ 牛/ 逼\n"
     ]
    }
   ],
   "source": [
    "import jieba\n",
    "seg_list = jieba.cut(\"#高校学生喊楼EDG# 给不搞电竞的睡觉人一条活路行吗,EDG牛逼\", cut_all=True)\n",
    "print(\"全模式: \" + \"/ \".join(seg_list))  # 全模式\n",
    "seg_list = jieba.cut(\"#高校学生喊楼EDG# 给不搞电竞的睡觉人一条活路行吗,EDG牛逼\", cut_all=False)\n",
    "print(\"精确模式: \" + \"/ \".join(seg_list))  # 精确模式\n",
    "seg_list = jieba.cut(\"#高校学生喊楼EDG# 给不搞电竞的睡觉人一条活路行吗,EDG牛逼\")  # 默认是精确模式\n",
    "print(\"默认模式: \" + \"/ \".join(seg_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffd46b66",
   "metadata": {},
   "source": [
    "## 第一步： 数据加载"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "d0979e76",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "38235\n"
     ]
    }
   ],
   "source": [
    "filter_set = set()\n",
    "with open('general.txt', encoding='utf-8') as ff:\n",
    "    for line in ff:\n",
    "        filter_set.add(line.strip())\n",
    "\n",
    "# 过滤词典(过滤掉没有太大意义的词语)\n",
    "print(len(filter_set))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "28251944",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['EDG DK S11 2021 总决赛 EDG vs DK 决胜局 EDG 先手 POKE 击杀 巨魔 巨龙 微博', '高校学生 喊楼 EDG', 'EDG flag 叫喊声', 'EDG 太困', '双龙 EDG DK S11 总决赛']\n",
      "['EDG DK S11 2021 总决赛 EDG vs DK 决胜局 EDG 先手 POKE 击杀 巨魔 巨龙 微博', '高校学生 喊楼 EDG', 'EDG flag 叫喊声', 'EDG 太困', '双龙 EDG DK S11 总决赛']\n"
     ]
    }
   ],
   "source": [
    "import jieba\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "x_dataset = []\n",
    "cnt = 0\n",
    "with open('edg.txt', encoding='utf-8') as f:\n",
    "    for line in f:\n",
    "        cnt += 1\n",
    "        if cnt == 1: continue # 过滤第一行表头\n",
    "        \n",
    "        arr = line.strip().split(\"\\t\")\n",
    "        seg_list = jieba.cut(arr[1]) # 截取句子\n",
    "        x_dataset.append(\" \".join([x for x in seg_list\n",
    "              if len(x) >= 2 and x not in filter_set])) # 过滤小于一个的词\n",
    "\n",
    "print(x_dataset[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "614247f1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EDG DK S11 2021 总决赛 EDG vs DK 决胜局 EDG 先手 POKE 击杀 巨魔 巨龙 微博\n",
      "<class 'scipy.sparse.csr.csr_matrix'>\n",
      "  (0, 16514)\t0.14234181742434854\n",
      "  (0, 16150)\t0.12137672631113564\n",
      "  (0, 15140)\t0.3709273312959591\n",
      "  (0, 15139)\t0.32394646150090195\n",
      "  (0, 8768)\t0.33731799277153035\n",
      "  (0, 8483)\t0.2718725310591102\n",
      "  (0, 7706)\t0.3794938361129448\n",
      "  (0, 3477)\t0.25912187758582755\n",
      "  (0, 3001)\t0.15023197932385315\n",
      "  (0, 2854)\t0.3905379816636643\n",
      "  (0, 1319)\t0.11579775779680118\n",
      "  (0, 1236)\t0.3263473203630643\n",
      "  (0, 238)\t0.16917158409553495\n"
     ]
    }
   ],
   "source": [
    "# 文本向量化\n",
    "tfidf_model = TfidfVectorizer().fit(x_dataset)\n",
    "\n",
    "# 得到tf-idf矩阵，稀疏矩阵表示法\n",
    "sparse_x = tfidf_model.transform(x_dataset)    \n",
    "print(x_dataset[0])\n",
    "\n",
    "print(type(sparse_x))\n",
    "print(sparse_x[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "835ce07c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 3 13  1 ...  0  0  0]\n",
      "the model is done!\n"
     ]
    }
   ],
   "source": [
    "# 模型训练\n",
    "# k-means++ 聚类中心尽可能的远\n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "y_pred = KMeans(n_clusters=15, init='k-means++', \n",
    "                random_state=520).fit_predict(sparse_x)\n",
    "\n",
    "print(y_pred) # x_date[0] => y_pred[0]\n",
    "print('the model is done!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "3322dd0e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the topic is:3\n",
      "[('EDG', 857), ('S11', 575), ('总决赛', 297), ('微博', 235), ('##', 208), ('edg', 145), ('DK', 137), ('s11', 130), ('2021', 80), ('LPL', 49)]\n",
      "\n",
      "\n",
      "the topic is:13\n",
      "[('EDG', 311), ('高校学生', 222), ('喊楼', 218), ('百鸟朝凤', 44), ('微博', 42), ('唢呐', 42), ('edg', 25), ('扰民', 24), ('11', 24), ('音乐学院', 23)]\n",
      "\n",
      "\n",
      "the topic is:1\n",
      "[('flag', 1055), ('EDG', 864), ('edg', 349), ('立下', 82), ('##', 49), ('微博', 42), ('朋友圈', 33), ('人不骗', 26), ('总决赛', 26), ('Flag', 25)]\n",
      "\n",
      "\n",
      "the topic is:0\n",
      "[('EDG', 17379), ('edg', 7393), ('##', 1091), ('朋友圈', 923), ('总决赛', 860), ('粉丝', 842), ('初雪', 723), ('flag', 647), ('LPL', 611), ('微博', 583)]\n",
      "\n",
      "\n",
      "the topic is:4\n",
      "[('EDG', 2707), ('edg', 977), ('Edg', 44), ('##', 39), ('...', 9), ('..', 4), ('电子竞技', 2), ('......', 2), ('0.5', 1), ('EdG', 1)]\n",
      "\n",
      "\n",
      "the topic is:14\n",
      "[('EDG', 2339), ('微博', 1963), ('edg', 583), ('##', 295), ('总决赛', 247), ('DK', 122), ('王俊凯', 116), ('flag', 94), ('S11', 94), ('11', 81)]\n",
      "\n",
      "\n",
      "the topic is:11\n",
      "[('哈哈哈', 721), ('EDG', 311), ('哈哈哈哈', 126), ('edg', 124), ('朋友圈', 48), ('flag', 22), ('微博', 16), ('牛批', 12), ('总决赛', 11), ('朱一龙', 11)]\n",
      "\n",
      "\n",
      "the topic is:9\n",
      "[('电竞', 1247), ('EDG', 705), ('就业率', 209), ('总决赛', 177), ('11', 160), ('朱一航', 158), ('微博', 153), ('战队', 105), ('62%', 100), ('2021', 84)]\n",
      "\n",
      "\n",
      "the topic is:8\n",
      "[('朋友圈', 962), ('EDG', 688), ('edg', 257), ('全是', 85), ('初雪', 73), ('DK', 42), ('发到', 41), ('微博', 32), ('醒来', 24), ('连发', 20)]\n",
      "\n",
      "\n",
      "the topic is:7\n",
      "[('EDG', 1563), ('战队', 1329), ('总决赛', 857), ('S11', 519), ('电子竞技', 434), ('DK', 351), ('LPL', 349), ('2021', 291), ('LCK', 234), ('微博', 197)]\n",
      "\n",
      "\n",
      "the topic is:5\n",
      "[('EDG', 366), ('LOL', 248), ('lol', 202), ('edg', 177), ('不玩', 41), ('##', 40), ('微博', 39), ('朋友圈', 32), ('手游', 29), ('战队', 25)]\n",
      "\n",
      "\n",
      "the topic is:6\n",
      "[('EDG', 518), ('有人', 420), ('朱一航', 188), ('edg', 158), ('持股', 45), ('总决赛', 42), ('超竞', 42), ('朋友圈', 39), ('电竞', 36), ('flag', 33)]\n",
      "\n",
      "\n",
      "the topic is:10\n",
      "[('刷屏', 375), ('EDG', 336), ('朋友圈', 225), ('edg', 91), ('初雪', 28), ('微博', 26), ('醒来', 13), ('玩游戏', 12), ('圈里', 9), ('打游戏', 9)]\n",
      "\n",
      "\n",
      "the topic is:2\n",
      "[('裸奔', 461), ('EDG', 368), ('edg', 227), ('吃屎', 180), ('跳楼', 119), ('flag', 80), ('洗头', 46), ('粉丝', 39), ('朋友圈', 38), ('室友', 33)]\n",
      "\n",
      "\n",
      "the topic is:12\n",
      "[('朱一龙', 634), ('EDG', 357), ('啊啊啊', 140), ('微博', 47), ('哈哈哈', 40), ('edg', 36), ('龙哥', 18), ('电竞', 18), ('哈哈哈哈', 16), ('2021', 12)]\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 统计不同主题top-n 关键词\n",
    "dict_topic = {}\n",
    "for pos in range(len(y_pred)):\n",
    "    set_w = x_dataset[pos].split(' ')\n",
    "    y = y_pred[pos] # 确定拿一个分类\n",
    "    if y not in dict_topic:dict_topic[y] = {}\n",
    "    \n",
    "    for w in set_w:\n",
    "        if w not in dict_topic[y]:dict_topic[y][w] = 0\n",
    "        dict_topic[y][w] += 1\n",
    "\n",
    "# 获取不同主题top-n 关键词\n",
    "for y in dict_topic:\n",
    "    print(\"the topic is:%s\" % y)\n",
    "    sorted_list = sorted(dict_topic[y].items(), key=lambda d: d[1], reverse=True)\n",
    "    print(sorted_list[:10])\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "690eee34",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}